g^EAST Browser - L2: (9) 1 nearS mask$ | US 5111286 A | Tag: S | Doc: 8/9 (SORTED) | Format : KWIC 



£lle t;! £dtt yiipv^Ioots Window ^ Help ^ I ^i? #• ^ -*f * / : ^ 



WW 



UJS. Patent May 5, 1992 Sheet 2 of 2 5,111,286 



& 



4 £ 



m en @ 

iii. 



si 



>5 



to 



In a 



will then be used in a 
subsequent processing operation 
for example to combine 
the image in the store 1 with 
another image. In this example, 
the image in the 
store 1 is fed via the LUT 2 
a store 3 as before. A 
processor 21 then 
accesses each pixel value in 
turn from the store 3 and 
applies this to a 
predetermined algorithm, 
simple case, the algorithm 
comprises a 

thresholding process in which 
the pixel value is compared with 
the threshold. 

If a pixel value exceeds the 
threshold then a control value 
"1" is generated 
while otherwise a control value 
"0" is generated. These values 
are then stored 
in a mask store 22. 
generate a so-called 
which can 

subsequently be used 
conventional manner 
sophisticated process 
the process-er—may apply—an 
algorithm for generating a 

our selective mask in 
which, for example, the pixel 
value accessed from the store 
is compared with 



This will 
"hard" mask 
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behI inugB notion between ft ptitof input tettw from 
which u output ^""t* it derived by motio n ou&peth 
vtcd tiiftipulittuiL A ph ueh t y of motion vectors ere 
gescnted far cacti pixel of the cctpct irnifte tad tt ii 
dttaBtned whether rapocdvt test Hock* of etch af 
the peir of ispvt itniflft, poin t ed to by etch of the plu- 
rality of moDOD maun. lie ptrtikOy oatdds their ro- 
■pectrve input fmtgr* if one or both of the tutbloc&i 
ha pertiilly oTO idc to rcrpecthx tnpm toitA the de- 
gree of wn station between the tMt bl mA» it detected 
by perfismnif ► flnt cocrdillus test on put* of the test 
blocks lytnj mtids their n 
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Detailed Description Text - DETX 
(15) : 

The pixel address generated 
by the address shifter 66 is 
then passed from 
the position detector 68 to the 
respective picture segment store 
60, which 

supplies as an. output a block of 
5. times. 5 pixels centred on the 
pixel pointed 

to by the motion vector under 
test. In this embodiment these 
blocks are 

supplied in full whether or not 

one or more rows or columns of 

pixels in the 

blocks lay outside the 

respective input image. The two 

blocks of pixels from 

the respective picture segment 

stores 60 are supplied to a 

block marcher 74 for 

further processing. In the 

block marcher 74 the luminance 

alues of pixels at 
corresponding positions in the 
two blocks of pixels are 
compared and an array 
of absolute rfiffprpprp vfl1in»3 f 
one value for each pixel 
position in the block, 
is generated and passed to a 
masker 76. ~ "~ 
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SinrQxriy, the twna ftw^ gm nyf fik n ft fctt tflft and fin- 

image being Comhiated by dob or more active lijto Heron 
typically dire c t e d by balnea to id rirrta to any i tnb fc n i 



wUch ate the thbjectof the iroJ^e, often aperiaa.The near 

beBl&ciea^by tn tedtt tight toonx. The to background 
rcaico ii the put cf (he background which cm not be 



jl kirr^^nn Tbo background is not ff Hrfr r r^ by the 3 t3tod^by tn eeri»c light icrac^Farrvo IR hafti with 



active foreground hghtfiig, The tcnni front lighting or front 
icurcei ere pffrt' t rt ^ " frip the optical axii of the imaging 

dftViCB tO iTlnrw j twf frm — th jart of (h* image. TTW cf 
<*»- H ti f « Hfm (i jt fiput if ffte line nf pfo^li ih«t f>pntw tha 10 

foreground tnd Nn^r^ "" 1 regioni of a<Hg£al image. Two 
digital imaging devieei have 9» itae virtual ipadal loca- 
tion when As Imaga takes by the devicei ct ftWteal A 
djgfei l Imaging device inchtOBi toy dwrfca which use* a 
CCD (charge -cocptai device) aich as. bants lioiad to, an u 
dec&onic camera or a essjeotder. 

digitally captured fffllgf ll replaced wbi a predetermined 
dby 



i!ty cf pUcli H etch ofto^ove^OT^tgu ender vtr ^ 

grciuaj'regiarj of ea taagTonLe sccnrTlacther worfj. die 
locations of the two IR fmtgei wffl vary between the 



TVfl TB rrftitpa wn titVen of the ibttb ee*ne under different 
[i||tti^^e o '''*^ t ' ww The flat ii e front IR innc* IR^-* 

The trnrftd i* » background IR image TRfc ffo™jp***'i with 

imaJE ^""fr ltl tt & foreground yd t ^"Hr^ rpi^irt 200 

, _ _ _ _ which is further broken down Into i nsar bftck^rousd r»^cn 

tki Between pixel! of the loafs. One af the Itg h ti m K 202 fcivfat abjeca trfrich ire efftcted by tcdve ligM 



Another Li the cosixtt mio. dcflns<l u tsz mlo of lncoxity 
yiJoo aS pixfli tl the urae, Lc. corretpoadmg, locxilon thu 
are compared befwtoj two inagei cf the unx kcej tikec 
tt the uiQD rttw 3 



totutes, tod l 



a, providai > mgfbi mode af cotnp«ri *o n, 

which am be cntlyxcd to delintaitr, tlx rungiuuiul tnd 1 
^tdCfnMsd tv^oeii ofllic i&ixjftt Hnwv^d, tflvenl pxbb* 

it of Ujfet intEsiity ii dkectly n^Aad lo tb* 
icSectEna of Ma objaa from which Hit Hgta lj being , 

xmj wHi ytt t hifh mflfrtimrt, then mettt of the Qjht 
tad tot to the object U reflected tad wtfltto fbr meuQiB- 
Howevti; & pcdbUnt occtm if lb* object bsat Qh> 
< flfp«»»A hut ■ bv ro^*ffwj>» tinea raly t «?n«1t mnM of ( 

the Ujnt incident 10 the object li lefleocd ccd xvitlibU far 

Id the jifiwit invention, the object of tbs bongs ii 
caxnlly a pmon. Thai, tb* httiltw of the tobject wffi 

jeasrally fofltrw the Hne of denwettioa leporciaf the t 
forejiouisd tod if* Vr CTin(l rcjiom of the tm*^ *. s ii 
kaown thet blonde htb? eaKbtU htgb te&eeu&ce nd bltck 
hilf f^M?*tt low icfleeusce to vUihtfi Sight. Thus when & 
pertoa fcxvfetg Uick hair it Ha tobjeci of in t"«t^ . the 

intensity of the rejected >1ifbte tight ^1^^ to the hinit x itjtcsi taring ta Icttsiby of "C which ii jreter the the 

hirfr will bs cJffiealt to aeaxare. tod uaicocpnble for loseottty "B" of the front lighting fa FIG. 2, for thii 

tt^eniity ccmp«ii»cni. On the other h*ail, lliht In the IR predated method, th« isXBzdVf of ths backfrvund lisktinf ii 

regtott of the light tprwniTn exhibfci higb itfactmce char- gttuer dstn the imeriifty of the frost H gfcriwg is thit the 

iceriitici fei both bloado and bltck hak Rathemoft, the reUoiondsi^t detailed in the foOowing Tkbb 1 w£] hoSil Uua. 

teatlti?lty of ccsnmescMy ividkble CCD* tocbdei the u The tsugei IRjqt Dtd TBt^o prefenaly taken wl&the 

near infrared range of ■ppnadraBeJy TOO tt> IPX ntaam- tune IR innjfcn t5B\k» in cm tsniporal procdnity, 

eieri. Thu»,aalvc light iomcc* la the i^-Lnf«r^ ooly by the ihuscr »pecd of the IR toagiei i&nx tat the 

" f to tits prwent tlmstsecenary ©»wr^ tbe^TelREeKiiiii. By alcngthe 
two ER images ai cist* tognther u poitihLa is time, prob- 

The pretent imtttloa tUowi an lnagetobetatetrBgird- k lemi cfeaweby the movemca of ±e rebjea ai objeia in 

leu of the bacKgiamd of die teens. Thtu, 1 pbotoboach or (he cetr beckgtouDd cm be avoided. More ipedfx*!h. as 

backdrop bchiad th» tubjsct Is aot repaired. However, • long utha movement of th* Use of ueuaaalion between the 

carelUl tcalysii of both the tea vc and esbies: lighting it in exporurti li not detectable by ±e hiuntn eye, then the 

eotei.R»rS»porpofeiof mecunentliwentlJ^ aieeaecin movement of the tubjea ii negHgible. TyatceDyr a tcari- 

be rtiiwctrf inffl ttstc region*; t foregflncti legios, a near t> mum nme dUTerenttal hetweea exposures of about VioCi of 

b*citnrcad regioi: and a far btcifromsd re^iot. The faro- ■ iztoad (i^. the ifcatttr tpeed of an decz^sic camcn) wiL 

" ice emure ne»jitihlc movetsca of the lufajcct. Of coarte. i.' the 



1 by active light w 
FlO, } tbowt the front fUmstatted IR image IR^ taken 
with finnt IR lightmg. Of twortc, arrbicnt {It. noa-adive) 
ER. forcgajuacj aod beckgronod IJgtaing wiU ii» be preim 

tsd die btdtgrouod 200. Only tmMffit IR light hating 
fsteciiry "A n It ndlectcd it varioui pixeii in the far hack* 

the lijhtiag of the faTbeckgrosal. For innaocc, if the icene 
It apeaoo ttasdiegm a lobby cf tlt^e building with aback 
wall 75 feet away, then the active IR lighting, Le. the front 
IR lighting. wi2 not Utoalmte the far beckgroond of the 
back wall. The front IR light, in nrymbini*frTi with the 
ambient IR light, ti capable of iflumlriatfcg ohjecti in the 
forcgjousid «n4 the sear background 303 ai a light 
isssmity *B* < wbkh 11 gicam then the ir Hint 

iiBtatity "A". 

FIO. 3 tbowt a background ni m w' M twt IR fcnagt IRao 
taken wfch no active front IR {Ifttndntrion Qjb, only tmbignt 
IR tTg^^ng ii preaent In the fiire ground 122-) and one or more 
backgroond IR llghu which bave been actJvced to Ommi- 

the near region 202 of the background 300, 

Ambient K li^ht ii rejected from pixcli ii: the foreground 
and fa background region* hiving an intensity of "A", and 
the background IR ligtutng conihtned with the t w ^i*"i rfl 
HgMing i* reflected frost piaelf in the near sackground 
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to a visible light image the jgi 
scene and the preselected 



background is used to 
replace the original background. 



Brief Summary Text - BSTX (9) : 

The present invention is 
useful in taking images for 
identification and 
other purposes without the 
requirement of a photobooth, 
regardless of the 
background of the image. The 
original background of the image 
is replaced with 
a preselected background. Two 
IR images with different 
intensities of IR 
illumination in the foreground 
and background regions of the 
scene, 

respectively, are compared for 
light intensity ditfcerencea *** 



betws 

c orresponding pixels of the two 

images to r'orm a mask 

dif f e rentiating between 

the foreground and background 

regions of the image. The mask 

is then applied 

to a visible light image of the 
scene and the preselected 
background is used to 
replace the original background. 
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x fnitnsiiy u the 
. el b fckctlfiw) in 
4 given cdul En tRj^ 
has the Mmt nrlam irrt en t ity ti the tame pixel ktcttxoB is 
IR M then that pixel it identified fs the mask u * btck- 

Ibc imaging devices 106 and 101 ipefeiably are one color 
CCD type end coe black and whiw (XD type (tlthongh two 
color CCD cj-pe* are acceptable) wii i jood outlay tele- 
v !«mm» t^« at ■ Ar**r+A fr»*tl Vngrti md filtered to fennel the 
spectral sensitivity u t dtaired spectral btsd. Compatible 
color video cameras 1116 "*1 108 tre preierrad whereby oat 
of th* furifw* with an IR pass, visible reject 

filter to be able to record En IR imige, All of the vtrUbles 
for taking i photograph each u the depth cf field, fool 
length, etc. tie easily established u atcesiajy by one of 
ordinary «V>*n ia imag ing science. 

In u experiment*! setup uied for letting the invention tt 
Fbliraid'e Imigin g Science Laboratory, e tingle ™«yrn g 
device wis used for taking both the IR tad voile tiahl 
images of i minTrccjttin. The tmsgtng device mmbtcd of a 
Phflipe CMS0O black A white NTSC format (W0x48Q 
pixels) CO) omtra with color •eptrtnosi ttsinj 
written 25 (red). 58 Ureca) *od 47B (bhic) altera. Red, 
green bhte images were iodividutlly recorded 
fj^rin^. Color balance wis adjusted using wrtuin setttrtl 
density filters endVbr **h«ngi^g the lunp volit£c for the three 
color filler exposures. The nrrtn included a Ofl Tnp rTir fft-4 
16 mm FL led with ■ 1 ami BG18 glue filter for IS. 
rejection and i wretten STB gel finer far visible Kgfa 
rejection. Digitization wis irypmpHihed tttfag- i Diti 
Translation DT55-LC frtme grabber wi!h 7 bits of cjuanti- 
mica. 

Dilcrcnt tize ipratarea eta be seed In the visible tod 
infrued cameras, lince the warping see will correct any 
misalignments between the visibi* and HI images. However, 
the ben system indodo visible and infrared camera* having 
i. TtM infrared 



faired embodiment, the foreground of the fri ^'g * is a pexsm 
having fail ffclnra for identification pjipusas >f *^ the 
nM^ppmimt **. lyrt mm t]ie ja the it 
Hie basic mask geaertooa method ts is outlined 
| in the block diagram af HG, 4. Asanas that the boot 
flhurinated IR fcuxe K/iJ), &e background flhcnfaawJ IR 
im«y Dl^ij), the visible light image Vjjp j) asd the pre- 
ddermiocd repliccincm badegronad B(ij) have aD been 
determined as described in Ibe above ecctions, where i and 
j are integers which re prese nt (he hrwiw«it«l and vertical 
orardinttts cf the images, respectively. A Cmeyuuiid pixel 
b» ox property W/U>-Dl»0j) ^ » background pixel hts 
the property of either Ulj^CkJ^) a I8/iJ><IB»(ij') 
dep eadi cg t^oa whether or cot the pixel it iOnninated by 
active lights, Ic t typical system, each pixel of ERjCj) and 
1R»(U) b rcprcicatcd ts 0aia/iJ)S255 or OSIR^y) 
S2S5, reepeotivery. Stibtractieg W^ij") from IH^ij) ia step 
400 yield* a difftreocc image DIFF(jj) wbere etcb pixel it 
3 represented ai -2555 DIFF(ij) 5+255. A sample fkS point 
* aiDffP[y) for Ij^U... 7 b shown betow. 
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DIFKiJ) is tanirued tn step 410 to form a biatrixed 
55 fnuge M/iJ) by cornpiring the raunrfciJ value of each 
pixel of DIFF to i predetemiried itrethold vthte 6, men 
letting t£ pixel vilues which tre greater than 8 to t logic 
high and all pixel values which are less thin or eqtul to 6 to 
a logic low. Tbit type of pixsl «*i»«iaj-«firt« li m«ttwn» ri. 
o etlly wrinea as: 



JC?y>S,2tXmU>«,a 



wbere 8 is s nredctennincd parameter which wfO be dis- 
cussed in further detail hereinafter in cacjuDctkm with 
c»if»TitW« for tbs nvyt^if ttftT^ ^TTt fon of step 430. 

The fallowing Sxfl bimriztd muge M/iJ) af the above 
difference image DIFF is Hbttrttrre for ij^U ... 7 when 



should have a largt apertui* to mat the background tn the 
isfrtied images will be bhsrred. To the extreme, the back- 
ground will appear wiff)** 7 * fot both I^ij^t. btiJ 

brighter when (he backgrouad is iUsmiaaieil The influx ci 
light can be costolled by osing a rrtasptreecy with sx 
appropriate trinmrtsswn rate. Most importarjtiy, the rrtfearei 
omtra used ahcrald respond seositively to tmail tignl " 
tiry ehartges when the tight is weak. 

The foreground flbmsinatiorj Car both the visible tad near 
IB. ranges in the test sytteen was provided by throe tungsten 
halogen Lowell Pro-LighU. model Pl-10 (125 watts, 3200K 
@ 120 volts) which were each placed between I tad 2 feet \f 
from the optical uds of the camera 300 and approximately 
IS feet from the subject. Exposure was controlled by 
rfitTujpg the lamp voltige. The background ilfeminstiori fbr 
the appropriate ER image r ecord sib provided by dace 
atagstea btbgen Lowell ToU-Ligba. mode] Tl-10 (500 
waits, 3200K @ 120 vohs) with bun doors cichtdisg the 

Mask Generation ' 

Mask generation is a crucial task in background replace- Note that a in 0 can cause a aignincant change m 
meal For the inventive background replacement method, a si the appetraace tad vihws of the bictrized image M^ij). 
mask is potratad tor teccrilsly daUngirithirig between lbs Cardrut ti» abov* biairtasd imagt for 8-5 with th* fol- 
rotcgronxjd and the backgrovrtd of an imaec. In one pre- towing bioarized image for 9-10. 
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Brief Summary Text - BSTX (6) : 

The parent case to this 
application discloses the 
general approach for 
replacing the background of an 
image by differentiating between 
two infrared 

(IR) light illuminated images to 
distinguish between the 
foreground and 

background of the corresponding 
visible light image. It 
specifically discloses 
a background replacement, method 
where two IR images with 
different intensities 
of IR illumination in the 
foreground and background 
regions of the scene, 
respectively, are compared for 
light intensity differe nces" 
between 



corresponding pixels of the two 
images to form a mask 
differentiating between 
the foreground and background 
regions of the image. The mask 
is then applied 

to a visible light image of the 
scene and the original 
background is replaced 
with a preselected background. 
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